Prostate-specific membrane antigen (PSMA) is an excellent biomarker for the early diagnosis of prostate cancer progression and metastasis. The most promising PSMA-targeted agents in the clinical phase are based on the Lys-urea-Glu motif, in which Lys and Glu are α-(l)-amino acids. In this study, we aimed to determine the effect of β- and γ-amino acids in the S1 pocket on the binding affinity for PSMA. We synthesized and evaluated the β- and γ-amino acid analogues with (S)- or (R)-configuration with keeping α-(l)-Glu as the S1'-binding pharmacophore. The structure-activity relationship studies identified that compound 13c, a β-amino acid analogue with (R)-configuration, exhibited the most potent PSMA inhibitory activity with an IC50 value of 3.97 nM. The X-ray crystal structure of PSMA in complex with 13c provided a mechanistic basis for the stereochemical preference of PSMA, which can guide the development of future PSMA inhibitors.